Repository navigation
charts/agentmesh: each router announces its own pod IP with several replicas - #883
Conversation
…licas; router.bootstrapTokenMaxUsages With router.replicaCount > 1 every router announced the shared Service name (/dns4/<release>-router/tcp/4501). A node dials that name, the Service sends it to either pod, and the libp2p handshake fails with "peer id mismatch" because a router is identified by its peer id. Each router now announces its own pod IP (downward API) when replicaCount > 1 and neither router.externalAddrs nor router.hostPort is set; a single replica is unchanged. A router keeps no credential across restarts and enrolls again on every start, so the bootstrap token (one use per replica) is used up by the first restart and the restarted router crash-loops with "Bootstrap token max usages exceeded". New value router.bootstrapTokenMaxUsages (default 0 = unchanged, one use per replica) lets the operator cover the restarts they expect; the README also points to router.useOidcToken as the way to avoid the token. Tested on kind (v1.37) with two routers and a node: the node attaches to both, and a deleted router pod re-enrolls and rejoins. Fixes google#881. Fixes google#882.
There was a problem hiding this comment.
Code Review
This pull request introduces support for running multiple router replicas (router.replicaCount > 1) in the Helm chart. Each router now announces its own pod IP instead of the shared Service name to avoid peer ID mismatches. Additionally, a new configuration option router.bootstrapTokenMaxUsages is introduced to allow overriding the default bootstrap token usage limit. Corresponding unit tests have been added to verify these behaviors. The review feedback suggests optimizing the router-statefulset.yaml template to only inject the POD_IP environment variable when it is actually needed (i.e., when replicaCount > 1 and neither externalAddrs nor hostPort is configured).
| {{- if or .Values.router.hostPort (gt (int .Values.router.replicaCount) 1) }} | ||
| env: | ||
| {{- if .Values.router.hostPort }} | ||
| # kubelet expands $(HOST_IP) in the args below. | ||
| - name: HOST_IP | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: status.hostIP | ||
| {{- end }} | ||
| {{- if gt (int .Values.router.replicaCount) 1 }} | ||
| - name: POD_IP | ||
| valueFrom: | ||
| fieldRef: | ||
| fieldPath: status.podIP | ||
| {{- end }} | ||
| {{- end }} |
There was a problem hiding this comment.
The POD_IP environment variable is injected into the container whenever replicaCount > 1. However, if externalAddrs or hostPort is configured, the --external-addr argument will use those instead of POD_IP, leaving POD_IP unused in the container environment.
We can refine the template condition so that POD_IP is only injected when it is actually going to be used (i.e., when replicaCount > 1 and neither externalAddrs nor hostPort is set).
{{- if or .Values.router.hostPort (and (gt (int .Values.router.replicaCount) 1) (not .Values.router.externalAddrs)) }}
env:
{{- if .Values.router.hostPort }}
# kubelet expands $(HOST_IP) in the args below.
- name: HOST_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
{{- end }}
{{- if and (gt (int .Values.router.replicaCount) 1) (not .Values.router.externalAddrs) (not .Values.router.hostPort) }}
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
{{- end }}
{{- end }}
kaisoz
left a comment
There was a problem hiding this comment.
Thanks for this helpful PR! I left a couple of comments. Please also review the failing CLA otherwise this PR won't merge
| -H "Content-Type: application/json" \ | ||
| -H "Authorization: Bearer ${ADMIN_TOKEN}" \ | ||
| -d '{"role": "mesh:role:router", "max_usages": {{ .Values.router.replicaCount | int }}}' \ | ||
| -d '{"role": "mesh:role:router", "max_usages": {{ .Values.router.bootstrapTokenMaxUsages | default .Values.router.replicaCount | int }}}' \ |
There was a problem hiding this comment.
Thanks for looking at #882 too! But I'm afraid this change alone doesn't fix it... The token expires after 24 h (the job sends no ttl_hours), and /enroll checks the expiry before the usages. So a router that restarts a day after the last install or upgrade still crash-loops.
The root cause is that the router enrolls on every start, because it keeps its credential only in memory. The node avoids this: it saves its credential and renews it with /refresh, which needs no token. The router could do the same:
- save its credential in
/data, next torouter.key, after each enrollment or refresh; - at start, load it and skip enrollment when it exists;
- let the existing
refreshExpiringCredentialrenew it./refreshaccepts an expired credential, so a router that was down for a while recovers too (unless the signing key rotated out meanwhile).
That's a Go change, so I'd do it in a separate PR. For this one, how about dropping bootstrapTokenMaxUsages and landing the #881 fix on its own?
| # one address per pod: a router's identity is its peer id, and the shared Service name sends a dial to whichever pod | ||
| # it likes ("peer id mismatch"). (Per-pod DNS would need serviceName to be a headless Service, which a StatefulSet | ||
| # cannot change on upgrade.) The pod IP changes on restart; the router announces the new one when it enrolls. | ||
| - "--external-addr=/ip4/$(POD_IP)/tcp/4501" |
There was a problem hiding this comment.
Nice fix! Although I think there is a simpler way to get the same result. If you leave out --external-addr, the router announces the addresses it listens on (advertisedAddrs in internal/router/router.go). With --listen=/ip4/0.0.0.0/..., that is the pod IP, for both TCP and QUIC. So this branch could be:
{{- else if gt (int .Values.router.replicaCount) 1 }}
# No --external-addr: each router announces the pod IP it listens on.
{{- else }}Then the POD_IP env var and the $podIP line are not needed. A bonus: QUIC gets announced too, while this line announces only TCP.
|
|
||
| ## Several routers (`router.replicaCount` > 1) | ||
|
|
||
| Each router announces its own pod IP (`--external-addr=/ip4/$(POD_IP)/tcp/4501`), because a router is identified by its peer id and the shared `-router` Service would send a dial to whichever pod it likes (the handshake then fails with "peer id mismatch"). With one replica, or with `router.externalAddrs` / `router.hostPort` set, nothing changes. The pod IP changes when a pod restarts; the router announces the new one when it enrolls. |
There was a problem hiding this comment.
nit: the router announces its address in its lease, which it sends on its first sync with the control plane after it starts and then renews. It does not announce the address when it enrolls. The same wording is in the comment in router-statefulset.yaml.
|
Thanks for the PR and for the test on kind. I agree with @kaisoz's review.
|
…addr so each router announces its own pod IP; drop bootstrapTokenMaxUsages
6b26c48 to
f6de1cd
Compare
|
Thanks |
Fixes #881.
With
router.replicaCount > 1every router announced the shared Service name, so a node dialing it landed on a random pod and the handshake failed withpeer id mismatch. In that case the chart now leaves out--external-addr, so each router announces the addresses it listens on: its own pod IP, for TCP and QUIC (suggested in review). A single replica,router.externalAddrsandrouter.hostPortare unchanged.README: new section on several routers, including that
replicaCountmust change through Helm (kubectl scaledoes not render the template again). Docs: one sentence in the Kubernetes guide, "Expose it". Helm-unittest cases for the several-replica case and forexternalAddrsstill winning.#882 (router restart and the bootstrap token) is left out of this PR, as agreed in review: the router will keep its credential on
/dataand renew it.Checked with
helm lintandhelm template(2 replicas: no--external-addr; 2 replicas withexternalAddrs: it wins). The unittest plugin was not run locally.